PySpark in Action: Python data analysis at scale MEAP V06 by Jonathan Rioux

Author:Jonathan Rioux [Rioux, Jonathan]
Language: eng
Format: epub, mobi, pdf
Publisher: Manning Publications Co.
Published: 0101-01-01T00:00:00+00:00

You can alias resulting columns, so you control their name and improve the robustness of your code.

Figure 5.8. A data frame arising from the application of the agg() method (aggregate function: F.sum() on Duration_seconds)

After the application of the aggregate function on our GroupedData object, we’re back with a data frame. We can then use the orderBy method to order the data by decreasing order of duration_total, our newly created column. We finish by showing 100 rows, which is more than what the data frame contains, so it shows everything.

Let’s select our commercials. Table 5.1 shows my picks.

Download

Copyright Disclaimer:
This site does not store any files on its server. We only index and link to content provided by other sites. Please contact the content providers to delete copyright contents if any and email us, we'll remove relevant links or contents immediately.